{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Single-tier language model example\n",
    "\n",
    "The following tutorial demonstrates how to utilize safekit's language modeling recurrent neural network to perform event-level anomaly detection. Unlike the aggregate autoencoder and its baselines, the language model is capable of detecting anomalous behavior at the event level. It accomplishes this by attempting to learn the syntax of log lines and the semantic relationships between individual fields in a log line. This allows the model to predict not only the likelihood of a network event, but also the likelihood of individual features appearing at given positions in the log line representation of that event."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import json\n",
    "import sys\n",
    "import os\n",
    "\n",
    "from safekit.batch import OnlineBatcher\n",
    "from safekit.graph_training_utils import ModelRunner, EarlyStop\n",
    "from safekit.tf_ops import lm_rnn\n",
    "from safekit.util import get_mask, Parser\n",
    "\n",
    "tf.set_random_seed(408)\n",
    "np.random.seed(408)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we'll define some hyperparameters for our model—these will be explained in greater detail as we go."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "layer_list = [10]\n",
    "lr = 1e-3\n",
    "embed_size = 20\n",
    "mb_size = 64\n",
    "\n",
    "maxbadcount = 10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we load the JSON file describing the specifications for the data.\n",
    "\n",
    "This JSON file describes a dictionary specifying the number of features in the input data; the categories corresponding to the features; whether the corresponding category is metadata, input, or output; and the indices which map these categories to specific features. This dictionary can later be used to ease interaction with the data when providing it as input to Tensorflow.\n",
    "\n",
    "`sentence_length` specifies a fixed sequence length over which our model will perform backpropagation through time, and `token_set_size` specifies the size of the vocabulary comprising all of the sequences—the former will be used to define the shape of the placeholders used for the features and targets, while the latter is used to define the shape of the embedding matrix used to map our categorical features to embedded representations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataspecs = json.load(open('../safekit/features/specs/lm/lanl_word_config.json', 'r'))\n",
    "sentence_length = dataspecs['sentence_length'] - 1\n",
    "token_set_size = dataspecs['token_set_size']\n",
    "\n",
    "x = tf.placeholder(tf.int32, [None, sentence_length])\n",
    "t = tf.placeholder(tf.int32, [None, sentence_length])\n",
    "ph_dict = {'x': x, 't': t}\n",
    "\n",
    "token_embed = tf.Variable(tf.truncated_normal([token_set_size, embed_size]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we define the recurrent neural network proper. A call to `lm_rnn` will instantiate all of the graph operations comprising our RNN and return a tuple of tensors: `token_losses`, which represents the token-wise losses over each input sequence; `h_states`, a sentence-length tensor comprised of the hidden states at each time step; and `final_h`, simply the hidden state at the last time step. For this call, we pass our input and output placeholders as well as our embedding matrix. We also provide a list of hidden layer sizes which determines the dimensionality of the hidden states at each time step—specifying more than one layer size will yield a stacked RNN architecture. The resulting model is a single-tiered RNN using Long Short Term Memory cells with a hidden dimensionality of 10.\n",
    "\n",
    "Finally, we define our losses over individual lines and over all lines by first averaging the feature-wise losses, then averaging these losses over an entire batch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "token_losses, h_states, final_h = lm_rnn(x, t, token_embed, layer_list)\n",
    "\n",
    "line_losses = tf.reduce_mean(token_losses, axis=1)\n",
    "avg_loss = tf.reduce_mean(line_losses)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To map losses back to our input features easily, we'll next define a function that we can call during the training loop that will write metadata and losses for each data point in the current minibatch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "outfile = open('results', 'w')\n",
    "outfile.write(\"batch line second day user red loss\\n\")\n",
    "\n",
    "def write_results(data_dict, loss, outfile, batch):\n",
    "    for n, s, d, u, r, l in zip(data_dict['line'].flatten().tolist(),\n",
    "                                data_dict['second'].flatten().tolist(),\n",
    "                                data_dict['day'].flatten().tolist(),\n",
    "                                data_dict['user'].flatten().tolist(),\n",
    "                                data_dict['red'].flatten().tolist(),\n",
    "                                loss.flatten().tolist()):\n",
    "        outfile.write('%s %s %s %s %s %s %r\\n' % (batch, int(n), int(s), int(d), int(u), int(r), l))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we instantiate a `ModelRunner` object, which provides a simple interface for interacting with the Tensorflow session. Instantiating this object will define the optimizer Tensorflow will use for gradient descent and initialize all of the variables in the Tensorflow graph. We can then use the `train_step` method on this object to perform an optimization step or the `eval` method to retrieve the values of arbitrary tensors in the graph.\n",
    "\n",
    "In order to record the losses for all of the features, we define a list `eval_tensors` that contains tensors whose values we want to retrieve during training. We'll provide this list to the `ModelRunner`'s `eval` method during the training loop to compute these tensors, then record their values with the `write_results` function defined previously."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = ModelRunner(avg_loss, ph_dict, learnrate=lr)\n",
    "\n",
    "eval_tensors = [avg_loss, line_losses]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For our experiments, we want to first train our model on a single day of user activity, evaluate the model's performance on the next day, then repeat this process for each day in the data. To ease this process, we'll define a function that will either train or evaluate our model over a single day of events.\n",
    "\n",
    "We first instantiate a batcher to divide the data into smaller portions. Since each day may contain a large number of events, we want to provide it to the model in small batches to avoid filling memory. Adjusting the minibatch size may also improve the model's performance. Here, we'll use a batch size of 64 data points, defined above as `mb_size`.\n",
    "\n",
    "We then define a stopping criteria for training using the `EarlyStop` object; if our model's performance doesn't improve after 10 training steps—defined above as `maxbadcount`—the `check_error` function we instantiate will return `False`, and training will be discontinued.\n",
    "\n",
    "In order to prepare data for training or evaluation, we manipulate raw batches from our batcher to construct a dictionary for Tensorflow that maps features to the placeholders used to feed data into the computational graph during training. We map the metadata features to their respective dictionary fields, define the upper range of our inputs and outputs with the `endx` and `endt` variables, then use these to select the appropriate features in the raw batch to determine our input and output.\n",
    "\n",
    "During training, we retrieve the losses for the current batch, then perform a training step to perform gradient descent over a single batch of inputs. This process repeats until either the batcher has reached the end of the input file, the stopping criteria has been met, or the model's error has diverged to infinity. During evaluation, we only retrieve the losses, then write these to our results file using `write_results`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def trainday(is_training, f):\n",
    "    batch_num = 0\n",
    "    data = OnlineBatcher('/home/hutch_research/data/lanl/char_feats/word_day_split/' + f, mb_size, delimiter=' ')\n",
    "    raw_batch = data.next_batch()\n",
    "    cur_loss = sys.float_info.max\n",
    "    check_error = EarlyStop(maxbadcount)\n",
    "    endx = raw_batch.shape[1] - 1\n",
    "    endt = raw_batch.shape[1]\n",
    "    training = check_error(raw_batch, cur_loss)\n",
    "    while training:\n",
    "        data_dict = {'line': raw_batch[:, 0], 'second': raw_batch[:, 1], \n",
    "                     'day': raw_batch[:, 2], 'user': raw_batch[:, 3], \n",
    "                     'red': raw_batch[:, 4], 'x': raw_batch[:, 5:endx],\n",
    "                     't': raw_batch[:, 6:endt]}\n",
    "\n",
    "        _, cur_loss, pointloss = model.train_step(data_dict, eval_tensors, update=is_training)\n",
    "        if not is_training:\n",
    "            write_results(data_dict, pointloss, outfile, batch_num)\n",
    "        batch_num += 1\n",
    "        \n",
    "        print('%s %s %s %s %s %s %r' % (raw_batch.shape[0], data_dict['line'][0],\n",
    "                                        data_dict['second'][0], ('fixed', 'update')[is_training],\n",
    "                                        f, data.index, cur_loss))\n",
    "        \n",
    "        raw_batch = data.next_batch()\n",
    "        training = check_error(raw_batch, cur_loss)\n",
    "        if training < 0:\n",
    "            exit(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For concision, we will train and evaluate our model on a small subset of our data. To train and evaluate over the entire data set, uncomment the lines following the current definition of `files`.\n",
    "\n",
    "Notice that if we use the entire data set, we reference a field in our data specifications called `weekend_days`. In our configuration files, we have specified a list of days in our data set which correspond to weekends. We want to exclude these days from training simply because they represent different patterns of user activity that may not match the distribution of user activities found during weekdays. To include these events in our analyses without affecting accuracy, another model can be trained on these events.|"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "files = dataspecs['test_files']\n",
    "\n",
    "# weekend_days = dataspecs['weekend_days']\n",
    "# files = [str(i) + '.txt' for i in range(dataspecs[\"num_days\"]) if i not in weekend_days]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we enter the training loop, which simply consists of two successive calls to `trainday`. The first call trains the model on the current day, and the second call evaluates the model on the following day."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "64 111.0 1.0 update 0head.txt 64 7.608037\n",
      "64 175.0 1.0 update 0head.txt 128 7.4731722\n",
      "64 239.0 1.0 update 0head.txt 192 7.493413\n",
      "64 773.0 2.0 update 0head.txt 256 7.3443861\n",
      "64 837.0 2.0 update 0head.txt 320 7.4136109\n",
      "64 1558.0 3.0 update 0head.txt 384 7.2962499\n",
      "64 1622.0 3.0 update 0head.txt 448 7.2749529\n",
      "64 2149.0 4.0 update 0head.txt 512 7.2419009\n",
      "64 2384.0 6.0 update 0head.txt 576 7.2701492\n",
      "64 2489.0 7.0 update 0head.txt 640 7.2029667\n",
      "64 2653.0 9.0 update 0head.txt 704 7.2183733\n",
      "64 2800.0 11.0 update 0head.txt 768 7.1504946\n",
      "64 2978.0 13.0 update 0head.txt 832 7.0741091\n",
      "64 3137.0 15.0 update 0head.txt 896 7.0784073\n",
      "64 3273.0 17.0 update 0head.txt 960 7.0569158\n",
      "64 3405.0 19.0 update 0head.txt 1024 7.0607281\n",
      "64 3591.0 22.0 update 0head.txt 1088 6.9573421\n",
      "64 3760.0 25.0 update 0head.txt 1152 6.9852099\n",
      "64 3947.0 27.0 update 0head.txt 1216 6.9511142\n",
      "64 4099.0 29.0 update 0head.txt 1280 6.8852801\n",
      "64 4256.0 31.0 update 0head.txt 1344 6.870347\n",
      "64 4376.0 32.0 update 0head.txt 1408 6.7629032\n",
      "64 4535.0 34.0 update 0head.txt 1472 6.7972283\n",
      "64 4765.0 37.0 update 0head.txt 1536 6.7118292\n",
      "64 4910.0 39.0 update 0head.txt 1600 6.7144222\n",
      "64 5083.0 41.0 update 0head.txt 1664 6.6843395\n",
      "64 5248.0 43.0 update 0head.txt 1728 6.6712413\n",
      "64 5436.0 45.0 update 0head.txt 1792 6.6916938\n",
      "64 5614.0 47.0 update 0head.txt 1856 6.5510864\n",
      "64 5785.0 49.0 update 0head.txt 1920 6.6234541\n",
      "64 5922.0 51.0 update 0head.txt 1984 6.5153809\n",
      "64 6135.0 54.0 update 0head.txt 2048 6.5792904\n",
      "64 6249.0 55.0 update 0head.txt 2112 6.5816317\n",
      "64 6413.0 57.0 update 0head.txt 2176 6.5243235\n",
      "64 6623.0 59.0 update 0head.txt 2240 6.6407309\n",
      "64 6823.0 61.0 update 0head.txt 2304 6.4981675\n",
      "64 6939.0 62.0 update 0head.txt 2368 6.4936333\n",
      "64 7117.0 64.0 update 0head.txt 2432 6.4755764\n",
      "64 7349.0 67.0 update 0head.txt 2496 6.4994678\n",
      "64 7502.0 69.0 update 0head.txt 2560 6.3575706\n",
      "64 7728.0 71.0 update 0head.txt 2624 6.3540773\n",
      "64 7942.0 73.0 update 0head.txt 2688 6.3188539\n",
      "64 8159.0 75.0 update 0head.txt 2752 6.2654562\n",
      "64 8384.0 77.0 update 0head.txt 2816 6.3844948\n",
      "64 8533.0 78.0 update 0head.txt 2880 6.265028\n",
      "64 8769.0 80.0 update 0head.txt 2944 6.2361555\n",
      "64 8972.0 82.0 update 0head.txt 3008 6.1505957\n",
      "64 9182.0 85.0 update 0head.txt 3072 6.1284156\n",
      "64 9469.0 89.0 update 0head.txt 3136 6.0370345\n",
      "64 9657.0 91.0 update 0head.txt 3200 6.0922546\n",
      "64 9895.0 94.0 update 0head.txt 3264 6.0984707\n",
      "64 10078.0 96.0 update 0head.txt 3328 6.0074816\n",
      "64 10325.0 99.0 update 0head.txt 3392 6.0837021\n",
      "64 10491.0 101.0 update 0head.txt 3456 6.1752014\n",
      "64 10702.0 104.0 update 0head.txt 3520 6.1028175\n",
      "64 10890.0 107.0 update 0head.txt 3584 6.048913\n",
      "64 11131.0 110.0 update 0head.txt 3648 6.0430012\n",
      "64 11362.0 113.0 update 0head.txt 3712 5.9952979\n",
      "64 11537.0 115.0 update 0head.txt 3776 6.0961485\n",
      "64 11703.0 117.0 update 0head.txt 3840 6.0769567\n",
      "64 11992.0 120.0 update 0head.txt 3904 6.0576992\n",
      "64 12119.0 121.0 update 0head.txt 3968 5.957283\n",
      "64 12309.0 123.0 update 0head.txt 4032 5.9075336\n",
      "64 12541.0 126.0 update 0head.txt 4096 5.9910965\n",
      "64 12754.0 128.0 update 0head.txt 4160 5.8898349\n",
      "64 12950.0 130.0 update 0head.txt 4224 5.925982\n",
      "64 13197.0 132.0 update 0head.txt 4288 5.8425465\n",
      "64 13514.0 135.0 update 0head.txt 4352 5.8077278\n",
      "64 13697.0 137.0 update 0head.txt 4416 5.9269948\n",
      "64 14053.0 141.0 update 0head.txt 4480 5.7065954\n",
      "64 14419.0 145.0 update 0head.txt 4544 5.8365998\n",
      "64 14701.0 148.0 update 0head.txt 4608 6.0161934\n",
      "64 14930.0 150.0 update 0head.txt 4672 5.8007035\n",
      "64 15316.0 153.0 update 0head.txt 4736 5.7946272\n",
      "64 15543.0 155.0 update 0head.txt 4800 5.829246\n",
      "64 15818.0 157.0 update 0head.txt 4864 5.8565021\n",
      "64 16103.0 159.0 update 0head.txt 4928 5.8217831\n",
      "64 16526.0 162.0 update 0head.txt 4992 5.7547245\n",
      "8 16873.0 165.0 update 0head.txt 5056 6.0375867\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Done Training. End of data stream."
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "64 15740885.0 86400.0 fixed 1head.txt 64 5.9215918\n",
      "64 15740949.0 86400.0 fixed 1head.txt 128 5.9538403\n",
      "64 15741172.0 86401.0 fixed 1head.txt 192 5.9176483\n",
      "64 15741236.0 86401.0 fixed 1head.txt 256 5.8954\n",
      "64 15741766.0 86402.0 fixed 1head.txt 320 5.939127\n",
      "64 15741830.0 86402.0 fixed 1head.txt 384 5.7233615\n",
      "64 15742960.0 86403.0 fixed 1head.txt 448 5.8417807\n",
      "64 15743024.0 86403.0 fixed 1head.txt 512 5.8680387\n",
      "64 15744269.0 86404.0 fixed 1head.txt 576 5.8846798\n",
      "64 15744592.0 86405.0 fixed 1head.txt 640 5.7462287\n",
      "64 15744859.0 86406.0 fixed 1head.txt 704 5.78092\n",
      "64 15745164.0 86408.0 fixed 1head.txt 768 5.8627472\n",
      "64 15745356.0 86409.0 fixed 1head.txt 832 5.8867254\n",
      "64 15745548.0 86410.0 fixed 1head.txt 896 5.8229733\n",
      "64 15745714.0 86411.0 fixed 1head.txt 960 5.7715893\n",
      "64 15745886.0 86412.0 fixed 1head.txt 1024 5.7244263\n",
      "64 15746128.0 86414.0 fixed 1head.txt 1088 5.8244162\n",
      "64 15746313.0 86415.0 fixed 1head.txt 1152 5.8875823\n",
      "64 15746478.0 86416.0 fixed 1head.txt 1216 5.797832\n",
      "64 15746742.0 86418.0 fixed 1head.txt 1280 5.8004789\n",
      "64 15746904.0 86419.0 fixed 1head.txt 1344 5.881063\n",
      "64 15747056.0 86420.0 fixed 1head.txt 1408 5.8041205\n",
      "64 15747212.0 86421.0 fixed 1head.txt 1472 5.7039266\n",
      "64 15747494.0 86423.0 fixed 1head.txt 1536 5.669641\n",
      "64 15747639.0 86424.0 fixed 1head.txt 1600 5.8056655\n",
      "64 15747826.0 86425.0 fixed 1head.txt 1664 5.7817025\n",
      "64 15747990.0 86426.0 fixed 1head.txt 1728 5.7214508\n",
      "64 15748281.0 86428.0 fixed 1head.txt 1792 5.8528781\n",
      "64 15748474.0 86429.0 fixed 1head.txt 1856 5.7366481\n",
      "64 15748662.0 86430.0 fixed 1head.txt 1920 5.8216867\n",
      "64 15748844.0 86431.0 fixed 1head.txt 1984 5.8340378\n",
      "64 15749015.0 86432.0 fixed 1head.txt 2048 5.830287\n",
      "64 15749209.0 86433.0 fixed 1head.txt 2112 5.7503939\n",
      "64 15749378.0 86434.0 fixed 1head.txt 2176 5.8499193\n",
      "64 15749500.0 86435.0 fixed 1head.txt 2240 5.7866879\n",
      "64 15749830.0 86437.0 fixed 1head.txt 2304 5.7226858\n",
      "64 15749993.0 86438.0 fixed 1head.txt 2368 5.8353796\n",
      "64 15750268.0 86440.0 fixed 1head.txt 2432 5.8964639\n",
      "64 15750427.0 86441.0 fixed 1head.txt 2496 5.8113041\n",
      "64 15750611.0 86442.0 fixed 1head.txt 2560 5.7728534\n",
      "64 15750915.0 86444.0 fixed 1head.txt 2624 5.8686485\n",
      "64 15751059.0 86445.0 fixed 1head.txt 2688 5.7239819\n",
      "64 15751359.0 86447.0 fixed 1head.txt 2752 5.6603479\n",
      "64 15751533.0 86448.0 fixed 1head.txt 2816 5.7879233\n",
      "64 15751686.0 86449.0 fixed 1head.txt 2880 5.8039427\n",
      "64 15751848.0 86450.0 fixed 1head.txt 2944 5.8774214\n",
      "64 15752116.0 86452.0 fixed 1head.txt 3008 5.6741657\n",
      "64 15752276.0 86453.0 fixed 1head.txt 3072 5.7477112\n",
      "64 15752553.0 86455.0 fixed 1head.txt 3136 5.9271889\n",
      "64 15752737.0 86456.0 fixed 1head.txt 3200 5.8510513\n",
      "64 15752901.0 86457.0 fixed 1head.txt 3264 6.0043092\n",
      "64 15753104.0 86458.0 fixed 1head.txt 3328 5.8682895\n",
      "64 15753276.0 86459.0 fixed 1head.txt 3392 5.8455105\n",
      "64 15753440.0 86460.0 fixed 1head.txt 3456 5.899909\n",
      "64 15753620.0 86461.0 fixed 1head.txt 3520 5.799686\n",
      "64 15753837.0 86462.0 fixed 1head.txt 3584 5.91641\n",
      "64 15753901.0 86462.0 fixed 1head.txt 3648 5.9752579\n",
      "64 15754218.0 86464.0 fixed 1head.txt 3712 5.9451942\n",
      "64 15754282.0 86464.0 fixed 1head.txt 3776 5.7435021\n",
      "64 15754618.0 86466.0 fixed 1head.txt 3840 5.9267778\n",
      "64 15754807.0 86467.0 fixed 1head.txt 3904 5.9025311\n",
      "64 15754986.0 86468.0 fixed 1head.txt 3968 6.0235977\n",
      "64 15755177.0 86469.0 fixed 1head.txt 4032 5.8517351\n",
      "64 15755357.0 86470.0 fixed 1head.txt 4096 5.9081087\n",
      "64 15755554.0 86471.0 fixed 1head.txt 4160 5.8318181\n",
      "64 15755739.0 86472.0 fixed 1head.txt 4224 5.784132\n",
      "64 15756080.0 86474.0 fixed 1head.txt 4288 5.8471637\n",
      "64 15756308.0 86475.0 fixed 1head.txt 4352 5.7810616\n",
      "64 15756639.0 86477.0 fixed 1head.txt 4416 5.7981181\n",
      "64 15756819.0 86478.0 fixed 1head.txt 4480 5.9305859\n",
      "64 15757008.0 86479.0 fixed 1head.txt 4544 5.9149141\n",
      "64 15757152.0 86480.0 fixed 1head.txt 4608 5.7188954\n",
      "64 15757437.0 86482.0 fixed 1head.txt 4672 5.7518101\n",
      "64 15757641.0 86483.0 fixed 1head.txt 4736 5.7720995\n",
      "64 15758001.0 86485.0 fixed 1head.txt 4800 5.9645042\n",
      "64 15758228.0 86486.0 fixed 1head.txt 4864 5.8697252\n",
      "64 15758436.0 86487.0 fixed 1head.txt 4928 5.926465\n",
      "64 15758636.0 86488.0 fixed 1head.txt 4992 5.9313288\n",
      "8 15758870.0 86489.0 fixed 1head.txt 5056 5.7042065\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Done Training. End of data stream."
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "64 15740885.0 86400.0 update 1head.txt 64 5.9215918\n",
      "64 15740949.0 86400.0 update 1head.txt 128 5.9392538\n",
      "64 15741172.0 86401.0 update 1head.txt 192 5.890296\n",
      "64 15741236.0 86401.0 update 1head.txt 256 5.852704\n",
      "64 15741766.0 86402.0 update 1head.txt 320 5.8825092\n",
      "64 15741830.0 86402.0 update 1head.txt 384 5.6522312\n",
      "64 15742960.0 86403.0 update 1head.txt 448 5.7559767\n",
      "64 15743024.0 86403.0 update 1head.txt 512 5.7725744\n",
      "64 15744269.0 86404.0 update 1head.txt 576 5.775579\n",
      "64 15744592.0 86405.0 update 1head.txt 640 5.6199107\n",
      "64 15744859.0 86406.0 update 1head.txt 704 5.6401129\n",
      "64 15745164.0 86408.0 update 1head.txt 768 5.7144151\n",
      "64 15745356.0 86409.0 update 1head.txt 832 5.7269464\n",
      "64 15745548.0 86410.0 update 1head.txt 896 5.6448817\n",
      "64 15745714.0 86411.0 update 1head.txt 960 5.5802975\n",
      "64 15745886.0 86412.0 update 1head.txt 1024 5.5255461\n",
      "64 15746128.0 86414.0 update 1head.txt 1088 5.606019\n",
      "64 15746313.0 86415.0 update 1head.txt 1152 5.6552658\n",
      "64 15746478.0 86416.0 update 1head.txt 1216 5.5495605\n",
      "64 15746742.0 86418.0 update 1head.txt 1280 5.5428858\n",
      "64 15746904.0 86419.0 update 1head.txt 1344 5.6147466\n",
      "64 15747056.0 86420.0 update 1head.txt 1408 5.5197039\n",
      "64 15747212.0 86421.0 update 1head.txt 1472 5.416563\n",
      "64 15747494.0 86423.0 update 1head.txt 1536 5.3666306\n",
      "64 15747639.0 86424.0 update 1head.txt 1600 5.4864864\n",
      "64 15747826.0 86425.0 update 1head.txt 1664 5.4568338\n",
      "64 15747990.0 86426.0 update 1head.txt 1728 5.3796449\n",
      "64 15748281.0 86428.0 update 1head.txt 1792 5.5055466\n",
      "64 15748474.0 86429.0 update 1head.txt 1856 5.362133\n",
      "64 15748662.0 86430.0 update 1head.txt 1920 5.448926\n",
      "64 15748844.0 86431.0 update 1head.txt 1984 5.4573965\n",
      "64 15749015.0 86432.0 update 1head.txt 2048 5.4363647\n",
      "64 15749209.0 86433.0 update 1head.txt 2112 5.3347988\n",
      "64 15749378.0 86434.0 update 1head.txt 2176 5.4288683\n",
      "64 15749500.0 86435.0 update 1head.txt 2240 5.3522801\n",
      "64 15749830.0 86437.0 update 1head.txt 2304 5.2788053\n",
      "64 15749993.0 86438.0 update 1head.txt 2368 5.3821602\n",
      "64 15750268.0 86440.0 update 1head.txt 2432 5.4393926\n",
      "64 15750427.0 86441.0 update 1head.txt 2496 5.3292675\n",
      "64 15750611.0 86442.0 update 1head.txt 2560 5.2803588\n",
      "64 15750915.0 86444.0 update 1head.txt 2624 5.3690023\n",
      "64 15751059.0 86445.0 update 1head.txt 2688 5.2261009\n",
      "64 15751359.0 86447.0 update 1head.txt 2752 5.1333179\n",
      "64 15751533.0 86448.0 update 1head.txt 2816 5.260581\n",
      "64 15751686.0 86449.0 update 1head.txt 2880 5.2661753\n",
      "64 15751848.0 86450.0 update 1head.txt 2944 5.3323822\n",
      "64 15752116.0 86452.0 update 1head.txt 3008 5.1005154\n",
      "64 15752276.0 86453.0 update 1head.txt 3072 5.1590786\n",
      "64 15752553.0 86455.0 update 1head.txt 3136 5.3644485\n",
      "64 15752737.0 86456.0 update 1head.txt 3200 5.2671905\n",
      "64 15752901.0 86457.0 update 1head.txt 3264 5.4300175\n",
      "64 15753104.0 86458.0 update 1head.txt 3328 5.2481422\n",
      "64 15753276.0 86459.0 update 1head.txt 3392 5.2222714\n",
      "64 15753440.0 86460.0 update 1head.txt 3456 5.2845478\n",
      "64 15753620.0 86461.0 update 1head.txt 3520 5.1773443\n",
      "64 15753837.0 86462.0 update 1head.txt 3584 5.2872629\n",
      "64 15753901.0 86462.0 update 1head.txt 3648 5.340085\n",
      "64 15754218.0 86464.0 update 1head.txt 3712 5.3005772\n",
      "64 15754282.0 86464.0 update 1head.txt 3776 5.0682311\n",
      "64 15754618.0 86466.0 update 1head.txt 3840 5.2326403\n",
      "64 15754807.0 86467.0 update 1head.txt 3904 5.2157192\n",
      "64 15754986.0 86468.0 update 1head.txt 3968 5.3379941\n",
      "64 15755177.0 86469.0 update 1head.txt 4032 5.1453543\n",
      "64 15755357.0 86470.0 update 1head.txt 4096 5.1733451\n",
      "64 15755554.0 86471.0 update 1head.txt 4160 5.0941672\n",
      "64 15755739.0 86472.0 update 1head.txt 4224 5.0607433\n",
      "64 15756080.0 86474.0 update 1head.txt 4288 5.0879831\n",
      "64 15756308.0 86475.0 update 1head.txt 4352 5.0342355\n",
      "64 15756639.0 86477.0 update 1head.txt 4416 5.0586109\n",
      "64 15756819.0 86478.0 update 1head.txt 4480 5.1715364\n",
      "64 15757008.0 86479.0 update 1head.txt 4544 5.1537275\n",
      "64 15757152.0 86480.0 update 1head.txt 4608 4.9285378\n",
      "64 15757437.0 86482.0 update 1head.txt 4672 4.9541712\n",
      "64 15757641.0 86483.0 update 1head.txt 4736 4.9644074\n",
      "64 15758001.0 86485.0 update 1head.txt 4800 5.1682501\n",
      "64 15758228.0 86486.0 update 1head.txt 4864 5.0511608\n",
      "64 15758436.0 86487.0 update 1head.txt 4928 5.0981541\n",
      "64 15758636.0 86488.0 update 1head.txt 4992 5.0900078\n",
      "8 15758870.0 86489.0 update 1head.txt 5056 4.9042935\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Done Training. End of data stream."
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "64 33303284.0 172800.0 fixed 2head.txt 64 5.3658361\n",
      "64 33303554.0 172801.0 fixed 2head.txt 128 5.2989731\n",
      "64 33303618.0 172801.0 fixed 2head.txt 192 5.1005902\n",
      "64 33304322.0 172802.0 fixed 2head.txt 256 5.2219648\n",
      "64 33304386.0 172802.0 fixed 2head.txt 320 5.1928158\n",
      "64 33305591.0 172803.0 fixed 2head.txt 384 5.257895\n",
      "64 33305655.0 172803.0 fixed 2head.txt 448 5.1073656\n",
      "64 33307036.0 172804.0 fixed 2head.txt 512 5.0622387\n",
      "64 33307100.0 172804.0 fixed 2head.txt 576 5.1791854\n",
      "64 33307306.0 172805.0 fixed 2head.txt 640 5.1440639\n",
      "64 33307636.0 172807.0 fixed 2head.txt 704 5.2225447\n",
      "64 33307700.0 172807.0 fixed 2head.txt 768 5.1567554\n",
      "64 33307944.0 172809.0 fixed 2head.txt 832 5.2424078\n",
      "64 33308094.0 172810.0 fixed 2head.txt 896 5.2141743\n",
      "64 33308295.0 172811.0 fixed 2head.txt 960 5.3878431\n",
      "64 33308426.0 172812.0 fixed 2head.txt 1024 5.094799\n",
      "64 33308663.0 172814.0 fixed 2head.txt 1088 5.1909528\n",
      "64 33308811.0 172815.0 fixed 2head.txt 1152 5.3374228\n",
      "64 33309020.0 172816.0 fixed 2head.txt 1216 5.1877556\n",
      "64 33309252.0 172817.0 fixed 2head.txt 1280 5.3042183\n",
      "64 33309461.0 172818.0 fixed 2head.txt 1344 5.3680315\n",
      "64 33309770.0 172820.0 fixed 2head.txt 1408 5.3148518\n",
      "64 33309990.0 172821.0 fixed 2head.txt 1472 5.3547783\n",
      "64 33310180.0 172822.0 fixed 2head.txt 1536 5.1936493\n",
      "64 33310393.0 172823.0 fixed 2head.txt 1600 5.1611204\n",
      "64 33310588.0 172824.0 fixed 2head.txt 1664 5.2133422\n",
      "64 33310813.0 172825.0 fixed 2head.txt 1728 5.1898479\n",
      "64 33311000.0 172826.0 fixed 2head.txt 1792 5.0969872\n",
      "64 33311217.0 172827.0 fixed 2head.txt 1856 5.3928194\n",
      "64 33311281.0 172827.0 fixed 2head.txt 1920 5.3400168\n",
      "64 33311445.0 172828.0 fixed 2head.txt 1984 5.2752399\n",
      "64 33311656.0 172829.0 fixed 2head.txt 2048 5.2166815\n",
      "64 33311844.0 172830.0 fixed 2head.txt 2112 5.1935391\n",
      "64 33312031.0 172831.0 fixed 2head.txt 2176 5.4086113\n",
      "64 33312292.0 172833.0 fixed 2head.txt 2240 5.2686853\n",
      "64 33312471.0 172834.0 fixed 2head.txt 2304 5.2384896\n",
      "64 33312650.0 172835.0 fixed 2head.txt 2368 5.4406233\n",
      "64 33312834.0 172836.0 fixed 2head.txt 2432 5.2461085\n",
      "64 33313002.0 172837.0 fixed 2head.txt 2496 5.2279139\n",
      "64 33313334.0 172839.0 fixed 2head.txt 2560 5.2823391\n",
      "64 33313511.0 172840.0 fixed 2head.txt 2624 5.2936883\n",
      "64 33313679.0 172841.0 fixed 2head.txt 2688 5.3466625\n",
      "64 33313838.0 172842.0 fixed 2head.txt 2752 5.2666893\n",
      "64 33314044.0 172843.0 fixed 2head.txt 2816 5.130785\n",
      "64 33314232.0 172844.0 fixed 2head.txt 2880 5.2147284\n",
      "64 33314491.0 172846.0 fixed 2head.txt 2944 5.2735906\n",
      "64 33314695.0 172847.0 fixed 2head.txt 3008 5.1918411\n",
      "64 33314983.0 172849.0 fixed 2head.txt 3072 5.2959223\n",
      "64 33315135.0 172850.0 fixed 2head.txt 3136 5.1151505\n",
      "64 33315324.0 172851.0 fixed 2head.txt 3200 5.4211307\n",
      "64 33315495.0 172852.0 fixed 2head.txt 3264 5.2560449\n",
      "64 33315710.0 172853.0 fixed 2head.txt 3328 5.0869799\n",
      "64 33316012.0 172855.0 fixed 2head.txt 3392 5.1635194\n",
      "64 33316183.0 172856.0 fixed 2head.txt 3456 5.2510548\n",
      "64 33316533.0 172858.0 fixed 2head.txt 3520 5.2242661\n",
      "64 33316747.0 172859.0 fixed 2head.txt 3584 5.4246845\n",
      "64 33316943.0 172860.0 fixed 2head.txt 3648 5.2669802\n",
      "64 33317140.0 172861.0 fixed 2head.txt 3712 5.277329\n",
      "64 33317295.0 172862.0 fixed 2head.txt 3776 5.455617\n",
      "64 33317503.0 172863.0 fixed 2head.txt 3840 5.2685261\n",
      "64 33317694.0 172864.0 fixed 2head.txt 3904 5.2825584\n",
      "64 33317978.0 172866.0 fixed 2head.txt 3968 5.4497023\n",
      "64 33318166.0 172867.0 fixed 2head.txt 4032 5.5404081\n",
      "64 33318347.0 172868.0 fixed 2head.txt 4096 5.2761469\n",
      "64 33318715.0 172870.0 fixed 2head.txt 4160 5.288794\n",
      "64 33318909.0 172871.0 fixed 2head.txt 4224 5.5018196\n",
      "64 33319084.0 172872.0 fixed 2head.txt 4288 5.1012011\n",
      "64 33319286.0 172873.0 fixed 2head.txt 4352 5.2988162\n",
      "64 33319623.0 172875.0 fixed 2head.txt 4416 5.409615\n",
      "64 33319896.0 172876.0 fixed 2head.txt 4480 5.3872242\n",
      "64 33320147.0 172877.0 fixed 2head.txt 4544 5.222887\n",
      "64 33320553.0 172879.0 fixed 2head.txt 4608 5.4052315\n",
      "64 33320778.0 172880.0 fixed 2head.txt 4672 5.4284472\n",
      "64 33321049.0 172881.0 fixed 2head.txt 4736 5.3590851\n",
      "64 33321113.0 172881.0 fixed 2head.txt 4800 5.2886515\n",
      "64 33321506.0 172883.0 fixed 2head.txt 4864 5.3202734\n",
      "64 33321682.0 172884.0 fixed 2head.txt 4928 5.2548733\n",
      "64 33321932.0 172885.0 fixed 2head.txt 4992 5.4764142\n",
      "8 33322150.0 172886.0 fixed 2head.txt 5056 5.3764067\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Done Training. End of data stream."
     ]
    }
   ],
   "source": [
    "for idx, f in enumerate(files[:-1]):\n",
    "    trainday(True, f)\n",
    "    trainday(False, files[idx + 1])\n",
    "outfile.close()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
